Brian PRO

CC's own CDX server does not index CC-NEWS at all.

that's why I'm doing this

I've tried all kinds of things on the hugging face space, I'm not going to try any new permutations of options. If I put 1 or -1 in max_open_files it still hangs in the same way (I don't think I literally tried 1). I reproduced locally using hf-mount and it hangs in the same way on the first query that tries to hit the db files.

here is a regenerated file with the new cdx-rocks-build command
https://huggingface.co/buckets/brian-learns/cdx-rocks-monthly

I've got this small demo index I'll use for testing with the hugging face. Once I get the space working, I'll see how large I can get the demo database
https://huggingface.co/buckets/brian-learns/cdx-rocks-demo

Common Crawl file naming conventions are part of their spec and documented on their dataset page on amazon, but the file sorting requirements seem worth documenting better in the dataset info.

/extent is only on the docker version, I'm not sure when I'll update the space. I started over in a new git repository and I think I need to do a force push, I might just build a new space. The current space hangs after about 40 consecutive hits because the files are in a bucket and I think the remote file system starts blocking up.

I'm not sure I agree about the two kinds of not found, I'm not sure how the index could know which the case is. https://index.commoncrawl.org does not have that, and https://archive.org/help/wayback_api.php is over their whole web archive (but does a whole different thing with robot.txt know as the Oakland Archive Policy). I don't think wayback tells if the URL got OAPed or if it was never crawled. I haven't looked at webrecorder's cdx index lookup, but that indexes your local WARC files from what I gather.

I wrote up a little fantasy spec for the cdx-rocks database definition https://github.com/brian-learns/cdx_rocks/wiki/database_definition -- I guess it could not hurt to include the information from the /extent endpoint formatted in the same json -- and it's not expensive to include, although to my mind it's redundant.

Looks like there might be some updates to the news crawler soon https://groups.google.com/g/common-crawl/c/SrAlK9l0lPA

I added /extent yesterday to the API and ccnget. It returns this from the one published index.

{
  "file_extent": 51101,
  "file_oldest": "crawl-data/CC-NEWS/2016/08/CC-NEWS-20160826124520-00000.warc.gz",
  "file_newest": "crawl-data/CC-NEWS/2026/07/CC-NEWS-20260731214950-00313.warc.gz"
}

The catalog file and the rocks db are implicitly versioned and need to match. I've only published one version really of the rocksdb, the first one was a test and I didn't announce it. It seems really rather neither here nor there if it gets recompressed or if the new warc_paths gets concatenated to the end, zstd should be able to be cat files together. If I publish more indexes I was thinking I'll create a little mini format for the index with the catalog in the root dir and the rocksdb in a subdir. Then it will just be one sync command for the whole directory with both artifacts.

Why would you make up WARC files that don't exist to inject in the middle? The WARC files are produced in chronological order and lexically sort in chronological order, as well the warc path files they publish have this same characteristic.

I don't understand what you are saying about the two type of misses. I think at a certain point news publishers started blocking the news crawler with robots.txt.

ccnget is just supposed to be dumb client for the API that also knows how to grab the content out of the WARC file with the byte offset method.

Thanks for your feedback. Please extend to me a generous reading, as I do to you. this is a tiny box in a tiny font and not easy to write in.

I think "!" is network byte order (big-endian), and the struct padding is only relevant to RAM

I think I'll change it to "!IQI".

Creating the cdxj files is what took awhile (almost a week on a c7g.2xlarge)-- I don't have any notes, but I don't think running the rocks index script took very long, maybe 30 to 60 minutes at the most?

I think I'll create two rocksdb -- one rocksdb with a subset of records that will fit in around 30G so it can run in the hugging face space. The second rocksdb will be the full size 75G and growing, but one has to run the docker to use it, and then uv run ccnget config set cdx-url http://0.0.0.0:7860/lookup. This is why I don't want ccnget to have to know about index versions.

Should probably pick a different port that the default for a hugging face space.

# --- RocksDB value format ---
# Each value is a 16-byte big-endian struct:
#   H (uint16) — WARC file ID (index into the catalog)
#   Q (uint64) — Byte offset within the WARC file
#   I (uint32) — Record length in bytes
# Total: 2 + 8 + 4 = 14 bytes (struct alignment pads to 16)
VALUE_FORMAT = "!HQI"
VALUE_SIZE = struct.calcsize(VALUE_FORMAT)

• max offset 1,073,731,048 -- 30 bits needed
• max length 1,048,157 -- 20 bits needed

seems like I could turn it to

#   I (uint32) — WARC file ID (index into the catalog)
#   Q (uint64) — Byte offset within the WARC file
#   I (uint32) — Record length in bytes
# Total: 4 + 8 + 4 = 16 bytes with no padding

and it would not really change the size of the db?

see https://huggingface.co/datasets/brian-learns/cdx-cc-news/tree/main -- rocksdb_from_cdxj.py was run on the original dataset, rocksdb_monthly.py added July. The scripts that builds the cdxj index files is in there too.

Yes, this approach is going to run out at 65,535 files, and will need a new format. I think that the file size does not need so many bytes as well.

I'm appending new lines to the end of the catalog as more WARC files are indexed -- it lists all the files that were processed to produce that dataset. Order in the catalog is the implicit warc file id in the rocks index.

I've cleaned up the shadow stuff so it only kicks in if needed (using a temp dir).

ccnget just talks to the /lookup endpoint, it has no concept of an "index version".

I'm not even sure if I'll create an index for Aug yet or not. That EC2 is spun down and I'm not going to start it up again unless I update the index.

I was thinking, if I do keep working on this, that the shadow directory should go in a docker tempfs, an you would just restart to pick up a new index.

If you are running all as the same user or don't have any permissions issues, ROCKS_SHADOW could actually point to your live index, the whole thing is just a workaround so I could start up the docker on my box. The hugging face space crashes after about 50 sequential requests, but it runs well on my DGX Spark.

I could 1) check permissions on ROCKS_DIR / rocksdict-config.json 1.a) if it's writable proceed 1.b) if it's not writable, do the symlink shadow hack. Then this detail does not leak to the user.

Thanks for taking a look, I didn't know about --delete on hf sync. I'm also not sure I closed or compacted the July index correctly, but it seems to work.

Next month if I index August I'll look at how the rocksdb gets closed out and make sure the sync deletes.

The cdx-rocks command is sort of a long story, it's an ugly hack but I couldn't out another way. setup_shadow inside of the main.py is an ugly hack too. But it's set up so it should be okay to run multiple times. You don't have to ever run the cdx-rocks command, the server will do it with setup_shadow if the shadow directory is empty.

For some reason rocksdict wants to write a file in the rocks db dir -- so the shadow dir is just symbolics links and one small json file that gets re-written every time it starts up.

ETA I tried to explain it better in the README https://github.com/brian-learns/cdx_rocks#local-setup

They have the code for the crawler up on the website, I've just processed all the WARC and created an index and a RocksDB so that I can make a tool to fetch articles from the WARC. The crawler follows robot.txt

https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html

https://github.com/brian-learns/ccnget