Back to Stream
ObservabilitySeptember 16, 202610% synthesized

The Best Incident Tool Might Be the One That Still Works When Everything Else Does Not

TermDevTools provides a lightweight, terminal-based alternative to the Kibana DevTools interface for interacting with Elasticsearch clusters. By removing browser and Kibana dependencies, it serves as a resilient, portable tool for troubleshooting in restricted or degraded infrastructure environments.

The Best Incident Tool Might Be the One That Still Works When Everything Else Does Not

Most Elasticsearch engineers have typed something like this during an incident:

bash curl -k -u user:password     https://cluster.example.com:9200/_cluster/health?pretty 
Then another request. Then a multiline JSON body. Then some
jq
. Then a certificate problem. Then somebody asks for shard allocation. Then you realize the Kibana instance you would normally use for DevTools is the thing you are currently troubleshooting. That scenario explains why TermDevTools is more interesting than its modest description suggests. The open-source project recreates much of the Elasticsearch Kibana DevTools experience inside a standalone terminal application. It provides a two-panel request and response interface, endpoint completion, execute-at-cursor behavior, formatted JSON, variables, request persistence, several authentication methods, result export, and other conveniences familiar to engineers who regularly work with Elasticsearch. Those features are useful. The larger idea is better: Operational tooling should survive the failure of the system normally used to operate it. Original project: https://github.com/gnocent/termdevtools ## Kibana Is Often the Best Elasticsearch Tool Until Kibana Is Part of the Incident Kibana DevTools is one of the most convenient ways to interact directly with Elasticsearch. An engineer can keep several requests in the editor, move between them, execute one, inspect formatted output, adjust a parameter, then repeat. During ordinary operations, that experience is excellent. During an outage, architecture matters more. Consider the dependency chain:
text Engineer    ↓ Browser    ↓ Kibana    ↓ Elasticsearch 
If Kibana is unavailable, the operator loses the usual control surface even when Elasticsearch itself remains reachable. The fallback becomes:
text Engineer    ↓ Terminal    ↓ curl    ↓ Elasticsearch 
That works. It is also where operational friction begins accumulating. TermDevTools inserts a lightweight interface into that second path:
text Engineer    ↓ Terminal    ↓ TermDevTools    ↓ Elasticsearch 
The browser and Kibana disappear from the dependency chain. That makes this less of a UI convenience and more of a resilience pattern. ## Administrative Tools Need Their Own Failure Domains Infrastructure teams spend significant time eliminating shared failure domains from production systems. Multiple availability zones. Replica shards. Redundant networking. Backup authentication paths. Separate management networks. Yet operational tooling sometimes gets treated differently. A team may depend heavily on the same web platform they use every day without asking: What do we use if this layer is unavailable? That question matters for Elasticsearch. You might still have:
text SSH access cluster network access API credentials client certificates Elasticsearch APIs 
yet lack Kibana. A terminal-native tool can operate inside that reduced environment. The distinction becomes particularly useful in restricted infrastructure where web access is intentionally limited. ## The Jump Host Test Is a Good Measure of Operational Software A useful test for infrastructure tooling is simple: Can I use it comfortably from a jump host at 2 a.m.? That environment strips away many assumptions. There may be no graphical desktop. No convenient browser. No direct laptop-to-cluster network route. Clipboard behavior may be awkward. Certificates may live on the host. Internet access may not exist. A static terminal binary becomes attractive very quickly. TermDevTools provides static builds for Linux amd64, Windows amd64, and Apple Silicon macOS, and the project notes that the Linux binary can be copied directly onto RHEL 8, 9, or 10 systems without a broader installation process. That portability changes where the tool can live. A copy can sit on:
text bastion host jump server administrative VM restricted enclave workstation engineer's laptop incident-response host 
The same interaction model can follow the operator into each environment. ## Ergonomics Become More Valuable as Pressure Rises There is a tendency to treat interface quality as cosmetic. During incidents, it is a reliability feature. An engineer under pressure is more likely to make mistakes. Long
curl
commands increase cognitive load. Escaped quotes increase cognitive load. Certificates and authentication flags increase cognitive load. Multiline JSON increases cognitive load. Parsing dense response output increases cognitive load. Every small annoyance competes with the actual problem the engineer is trying to solve. TermDevTools restores several conveniences that normally disappear once Kibana does:
text request editor formatted JSON line numbers endpoint completion search saved requests variables response export 
None of those capabilities is revolutionary. Together, they reduce mental overhead. That can matter more during an incident than another feature added to the production platform. ## Execute-at-Cursor Is More Valuable Than It Sounds One small feature captures this philosophy well. Several Elasticsearch requests can live in the same editor, separated by blank lines, and the operator executes the request beneath the cursor. That mirrors the way many engineers already use Kibana DevTools. During an investigation, a scratchpad might contain:
text GET _cluster/health  GET _cat/nodes?v  GET _cat/shards?v&s=state  GET _cluster/allocation/explain  GET _nodes/stats 
The engineer moves between them as the incident changes. That is more efficient than repeatedly reconstructing shell commands. The tool is preserving an investigative workflow, not simply sending HTTP requests.
The Best Incident Tool Might Be the One That Still Works When Everything Else Does Not

Saved Requests Turn the Terminal Into an Operational Notebook TermDevTools automatically saves work per cluster and per user. That creates another useful behavior. The request editor becomes a lightweight operational notebook. An engineer can maintain familiar diagnostic queries for different environments:
text production staging SIPR NIPR customer cluster lab 
Each cluster can retain its own working set. That is valuable during recurring incidents. Engineers rarely start every investigation from zero. They build up a personal library of useful API calls. Operational tooling becomes more valuable when it remembers those habits. ## Variables Can Reduce Dangerous Copy-Paste Editing Reusable
${name}
variables provide another small but practical improvement. Consider a request involving an index, node, or repository name. Without variables, an engineer may duplicate a request several times and manually modify each copy. That creates opportunities for mistakes. A variable-driven request can retain the structure:
text GET ${index}/_search {   "query": {     "match_all": {}   } } 
and let the environment-specific value live elsewhere. This is not sophisticated configuration management. It does reduce repetitive editing. During high-pressure troubleshooting, reducing repetitive editing is worthwhile. ## Copy as curl Creates an Escape Hatch From the Escape Hatch One particularly sensible feature is the ability to convert the active request into an equivalent
curl
command with secrets redacted. That means the terminal UI does not trap the operator inside its own abstraction. You can create and test a request comfortably, then export it into a universal format. That supports several workflows:
text share command with teammate paste into incident notes put command in runbook test outside TermDevTools convert into automation 
Good infrastructure tooling should make it easy to leave. The tool should reduce friction without becoming a new dependency. ## Security Choices Matter More for Administrative Utilities Any tool capable of sending arbitrary Elasticsearch requests deserves scrutiny. TermDevTools supports Basic authentication, API keys, and client certificates. The project states that passwords, API key secrets, and private-key passphrases are not persisted to its configuration. TLS certificate verification remains enabled by default, and local configuration files use restrictive filesystem permissions. Those choices fit the role. An incident tool should not solve one operational problem by quietly creating a credential-storage problem. The project has gone through a code and dependency security review according to its documentation, though the author is equally clear that the software is community-maintained, supplied without warranty, and places responsibility for production actions on the operator. That last point matters. ## Convenience Must Not Hide Destructive Capability A polished interface can make dangerous commands feel ordinary. Elasticsearch APIs can delete indexes, alter mappings, modify lifecycle policies, change cluster settings, or trigger other high-impact actions. TermDevTools deliberately executes what the operator writes. That means the safest mental model is the same one engineers should apply to Kibana DevTools:
text Readable interface ≠ safe command 
The interface lowers interaction friction. It does not lower the consequences of the request. Break-glass tooling needs experienced operators. ## There Is a Broader Lesson for Air-Gapped and Restricted Environments Modern software increasingly assumes permanent browser access and cloud connectivity. Many important environments do not work that way. Government systems. Defense networks. Regulated infrastructure. Industrial systems. Restricted management networks. Offline incident environments. In those places, the useful question is often not: What is the richest interface available? It is: What functionality survives the environmental constraints? A self-contained terminal application has several natural advantages:
text no browser requirement no Kibana requirement no SaaS dependency no external service requirement direct cluster communication 
That simplicity can be a feature. ## Static Binaries Still Have an Important Place The technology industry spends a lot of time building distributed platforms. There remains something deeply useful about one executable. A static binary is easy to reason about. Copy it. Run it. Connect to the endpoint. No background server. No container stack. No database. No browser. No separate frontend. No orchestration layer. For ordinary business applications, this may feel limited. For recovery tooling, it can be ideal. The fewer things required to operate the recovery tool, the fewer things that can stop you from using it. ## Incident Tooling Should Degrade Gracefully This suggests a broader engineering principle. Every important operational capability should have layers. For Elasticsearch, that might look like:
text Normal operations      ↓ Kibana  Reduced operations      ↓ TermDevTools  Lowest-level fallback      ↓ curl / REST API 
Each level sacrifices some convenience. Each removes dependencies. That is graceful degradation applied to administration. Teams already design applications this way. Operational tooling deserves the same thinking. ## There Is Value in Preserving Muscle Memory One reason TermDevTools is interesting is that it does not invent a completely different Elasticsearch interaction model. It mirrors the familiar DevTools style:
text METHOD endpoint JSON body 
That matters. Emergency tooling should not require engineers to learn a new interface during the emergency. The closer the fallback resembles everyday workflows, the less switching cost exists when teams need it. This is a subtle form of resilience. The backup tool preserves human muscle memory. ## Runbooks Could Include the Tool, Not Depend on It A useful operational pattern would be to maintain investigation snippets that work in both TermDevTools and Kibana DevTools. For example:
text GET _cluster/health  GET _cat/allocation?v  GET _cat/shards?v&s=state  GET _cluster/settings?include_defaults=true  GET _nodes/stats/fs,jvm,process 
Those commands remain Elasticsearch API calls. The interface becomes interchangeable. That means the runbook depends on Elasticsearch, not the client used to reach it. This is exactly what good operational abstraction should accomplish. ## The Best Recovery Tool Has Very Few Dependencies Think about common emergency tools.
ssh
curl
dig
openssl
tcpdump
jq
They remain popular partly for the same reason. They work in constrained environments. TermDevTools fits into that tradition. It takes one narrow administrative task, working directly with Elasticsearch APIs, and makes it much more comfortable without creating a large surrounding platform. That narrowness is part of its appeal. ## This Is Where Small Open-Source Tools Often Matter Most Large commercial products understandably optimize for common workflows. Small community tools can focus on narrow operational gaps. "No Kibana, but I still need DevTools." That problem is too specific to justify an entire platform. It is perfect for a focused utility. Projects like this often become valuable inside engineering teams precisely since someone encountered an annoying real-world constraint and built the tool they wished existed. ## The Bigger Lesson TermDevTools can be described in one sentence: Kibana DevTools-style Elasticsearch access from a terminal. That description is correct. The more useful interpretation is: It gives Elasticsearch operators a second control surface with a different dependency chain. That matters. Kibana can fail. Browsers can be unavailable. Network zones can prevent normal access. Jump hosts can become the only route into production. Restricted environments can remove conveniences engineers normally assume. In those moments, the difference between knowing Elasticsearch APIs and being able to work with them comfortably can affect how quickly an incident is diagnosed. TermDevTools does not replace Kibana. That is exactly why it is interesting. A resilient operational environment should not depend on one interface. Sometimes the most valuable tool in the stack is the one you barely notice during normal operations, then become extremely glad to have when the normal path disappears. ### Original Source gnocent, TermDevTools, GitHub. https://github.com/gnocent/termdevtools