Visualise Cloudflare Edge Logs with Grafana

Optimizely currently has an open beta for DXP customers to get access to their Cloudflare Edge logs. This beta has been around for a while, but I haven't seen that much posted about it, perhaps because edge logs can be kind of boring?

Well, not if we visualise them with fancy dashboards!

Tech Stack

How this all works (at least during the beta phase) is that the Cloudflare edge logs are pushed to an Azure Blob Storage container owned by Optimizely, which you get a SAS link for. What you do with the logs from that storage container is up to you.

I've chosen to use Grafana as it's a popular choice among the homelab crowd (guilty) because it makes it easy to visually represent pretty much anything. In order to get Grafana to "understand" our logs they need to go to Loki first, which you can think of as the underlying database engine.

The last thing we need is to get the actual log files from the Azure storage container into Loki. There may be ways to do that automatically, such as with an ingestion pipeline from Blob Storage to Azure Log Analytics, but I like building my own stuff so I've done that instead.

I've wrapped everything we need in a Docker stack to make working with it super easy.

Setup

Getting this going locally should be fairly simple.

Prerequisites

1. Get access to the edge logs beta by applying for it.
2. Install Docker.
3. Once approved for the beta, go to the PaaS portal and create an API key for the approved project. Make sure to select "Edge Logs" as the required permission.

Project Setup

1. Clone the git repo.
2. Create an `/.env` file based on my example.env, replace with your values where needed. Consider changing the default Grafana username and password while you are at it.
3. Open a terminal in the project directory and start the docker stack with `docker compose up --build -d`
4. Grafana should now be running at http://localhost:3000
5. Login as user / pass configured, or default admin/admin

You should have the stack running in Docker.

If you need to debug anything with the log ingestion, check the container for it.

Once done, shut down with `docker compose down` in the same project terminal to bring the stack down.

Adding a Dashboard

You can import the dashboard from my example above with the Grafana UI using the example dashboard JSON file. Give it a couple of minutes on initial startup to get enough data to fill all the panels.

Useful Links

If you want to explore the edge logs more the Cloudflare Edge Logs docs are a good reference for what the values mean so you can create your own dashboard panels from there.

If you run into any issues with access to the logs you may need to check the documentation from Optimizely. I've taken care of what the EpiCloud module does in my code to get the SAS link automatically as that's just easier to have it auto-refresh the SAS link when it expires.

And in case you want to contribute or raise issues: https://github.com/jacobpretorius/Opti.Edge.Logs.Grafana 

Conclusion

I'm really interested to see what visualisations people come up with. So far, I've found the Traffic Type and CF Block Actions to be very nice as those aren't things we're usually exposed to.

Please share your custom dashboards and panels with us 😇

Presentation

I spoke about this at the London dev meetup, watch the talk below.

https://youtu.be/yqtleljncDg?t=5159