Extr · A little order from a lot of text.

A logfile can be perfectly ordinary right up to the moment it is 2.5 gigabytes. 1 1. That number mattered because I needed a workflow that stayed fast enough to finish before losing the context in a night of incident handling.

After an issue, that was sometimes what I had. A hundred megabytes, 2.5 gigabytes, occasionally more. The question I needed to answer was usually small: what happened, roughly when, and how often. The answer was in there, wrapped in a few million lines that had never been organised for anybody’s convenience.

The obvious move was to send the file somewhere large enough to cope with it. That is where I always hesitated, because sending it somewhere meant keeping it. An index, a retention policy, a bill, a thing to explain to somebody a year later.

Which leaves a question that sounds trivial and is not:

Where do you put a file you do not want to keep?

What the large platform got right

At the time, I was working with a large data-analysis platform. It gathered enormous amounts of information and let us search through material that had never been carefully organised in the first place. What fascinated me was not the scale by itself. It was the moment when a small pattern could reach into a wall of unstructured text and pull out something meaningful.

A timestamp became time. A name became a group. A repeated fragment became a count. The logfile had looked like noise, but the noise had a shape.

This was the age when Big Data seemed to grow larger every time somebody said the words. Systems collected more, stored more, and measured how much they ingested. The number itself could become something to flaunt: look at all the ingestion we do. There was something quietly absurd in that. Data is worth something once you can make sense of it. Storing it is the invoice, not the insight.

The part I actually wanted was portable. The machinery around it was not.

One file, one question

My problem in the day to day was much smaller. I had one file. I had one question. I did not need an empire of stored information around it.

So I wrote down what I did not want: no long installation, no collection service, no dependencies to maintain, and no reason to send the file anywhere. What was left was a fairly short list of things I did want, and one observation. The browser on my machine was already installed, already fast at reading text, and already sandboxed. I could borrow it for the length of one investigation and then give it back.

Nine years ago, that simple need became Extr.

Point at a line and name what matters

The idea was simple enough to explain without a manual. Open a logfile, find a line that contains something useful, and point out the parts that matter. Give those parts names. Extr can then look for the same shape throughout the file and turn it into something you can search, count, compare, and eventually put into a report.

In practice the pointing is the interesting part. You browse 200 lines at a time, pick a line that looks representative, and select the value you care about. Highlight a timestamp and Extr offers to add it as a datetime field. Highlight an identifier and you can name it, then keep or edit the regex it proposes. When the file is a format somebody has already standardised, JSON Lines, WildFly and JBoss server logs, syslog, access logs, or just repeated key and value pairs, it recognises the shape and offers a set of fields to review. 2 2. The suggestion is always reviewable rather than automatic. A format guess that silently mislabels a field is worse than no guess, because you will trust the resulting count.

Once the lines have names, questions get short:

search "rollback" level=ERROR
| top 10 service

timechart time=datetime span=5m count

That is the borrowed idea, stripped down to the one piece I missed. A pipeline reads left to right, each stage narrows or reshapes what the previous stage produced, and the awkward cases fall back to a raw pattern with rex field=message /id=(?<request_id>\d+)/i.

The honest limitation is worth stating, because it shapes how the tool feels to use. Changes update the 200-line preview immediately, which makes experimenting cheap. Whole-file totals only change when you deliberately scan the whole file. Fast guessing, then slow confirmation, instead of waiting several minutes to learn that your regex had a typo in it.

What it refuses to remember

The file remains on your computer. Client-side parsing the browser reads the file directly and extracts matches into in-memory structures, then renders derived tables and counts without uploading content does the work, in a local worker, so a long scan does not freeze the page.

You can save a recipe and reload it next time the same system misbehaves. What gets saved is the rules and the display settings. Not the source text, not the extracted values, not the results. Running it again re-reads the file from disk, because there is no index to go stale.

For a long time I would have said the useful feature was the extraction. It is not. It is the forgetting.

That is the answer to the question at the top. The file did not need to go anywhere. It could stay where it landed, give up the one answer I needed, and then be deleted without ceremony, no retention policy, no index to rebuild, no monthly line item, and nothing to explain to anybody a year later. When the investigation is over, the tool does not need to keep collecting anything in the background. It can simply wait until the next difficult file arrives.

Sometimes a large file is just a temporary obstacle between you and one small answer. It helps to have something that finds the story inside, and then lets you close the file for good.