FALCON - Fetch and Log Continuous Observations of Networks

FALCON is a tool for continuously fetching websites and logging performance metrics like page load time, element rendering, and more. It provides a flexible way to monitor websites and PNG file performance over time, with customizable sleep intervals and automatic logging of metrics.

Features

• Continuous Website and PNG Fetching: Continuously fetches a list of websites or PNG files, logs the time it takes to load the entire page, and allows for the waiting of specific elements to fully render.
• Customizable Sleep Intervals: Allows users to define a minimum and maximum sleep time between requests to prevent constant polling.
• Save and Load Configuration: Automatically saves the user’s input (URLs, specific elements to wait for, sleep intervals) into a configuration file (config.json) and loads it on startup.
• Metrics Collection: Logs the time for a full page load and specific elements to become visible, and provides detailed metrics like minimum, maximum, and average load times for each URL.
• CSV Logging: Saves the performance data in a CSV file (fetch_times.csv), making it easy to analyze trends and performance over time.

How It Works

Fetch and Load Mechanism

FALCON uses Chromedp to load websites and PNG files, and it operates by:

1. Fetching PNG Files:

   • For PNG files, the fetch mechanism simply requests the file and logs the time it took to fully receive it.

2. Fetching Websites:

   • For websites, FALCON waits for the entire page to load, including all static assets (such as images, CSS, JavaScript) by waiting for the load event.
   • FALCON also waits for a user-specified element to become visible. This ensures that dynamic content (like JavaScript-rendered elements) is fully loaded before calculating the total time.
   • FALCON adds a small buffer by waiting for network idle conditions to ensure all asynchronous loading is completed.

Collected Metrics

1. Full Page Load Time:

   • The time from the start of the navigation to the point where the entire page (including all images, stylesheets, and scripts) is loaded.

2. Element Render Time:

   • The time it takes for a specific element (e.g., footer, main content) to become visible. This can be useful for monitoring dynamically loaded content.

3. Sleep Between Fetches:

   • A user-specified min and max sleep interval is used to control how often FALCON fetches the URLs, ensuring it doesn’t overload the server with constant requests.

4. Minimum, Maximum, and Average Load Time:

   • FALCON calculates the minimum, maximum, and average load time for each URL, providing insight into the performance over multiple fetch attempts.

Installation

Prerequisites

• Go (version 1.19 or higher)
• Chromedp: FALCON uses Chromedp, a Go library for driving headless Chrome.

You can install Chromedp by running:

go get -u github.com/chromedp/chromedp

Build and Run

To build and run FALCON from source:

git clone https://gitlab.com/<your-username>/<your-repo>.git
cd <your-repo>
go build -o falcon
./falcon

For specific platforms:

• Windows: GOOS=windows GOARCH=amd64 go build -o falcon.exe
• Linux: GOOS=linux GOARCH=amd64 go build -o falcon
• Mac: GOOS=darwin GOARCH=amd64 go build -o falcon

Usage

1. Enter URLs: Provide a list of URLs (one per line) in the URL input field.
2. Enter CSS Selector: Specify a CSS selector (e.g., #footer, .content) that should be visible before calculating the load time.
3. Set Sleep Intervals: Define the minimum and maximum sleep intervals (in seconds) between fetch attempts.
4. Start: Click the Start button to begin fetching the URLs continuously.
5. Stop: Click the Stop button to halt the fetching process.

Configuration File

FALCON automatically saves your input into a configuration file (config.json) when you start fetching. On subsequent runs, this configuration will be loaded automatically, pre-filling the input fields.

The config.json file has the following structure:

{
  "urls": ["https://example.com", "https://another.com"],
  "element": "#footer",
  "min_interval": 5,
  "max_interval": 10
}

You can manually edit this file to change the URLs, element, or intervals.

Metrics Example

The following metrics are collected for each URL:

• Min Load Time: The shortest time taken to load the full page.
• Max Load Time: The longest time taken to load the full page.
• Average Load Time: The average time across all fetches.

The CSV file (fetch_times.csv) will contain logs like this:

Timestamp,URL,Load,TTFB,LCP
2024-10-05T12:34:56Z,https://example.com,2.34s,1.23s,1.45s
2024-10-05T12:36:00Z,https://example.com,2.11s,1.12s,1.34s
2024-10-05T12:37:30Z,https://another.com,1.45s,0.89s,1.12s

Download

You can download the latest release for your platform below:

• Windows version
• macOS version
• Linux version

License

This project is licensed under the MIT License - see the LICENSE file for details.