diff --git a/.coderabbit.yaml b/.coderabbit.yaml index 5058aef..9085e7d 100644 --- a/.coderabbit.yaml +++ b/.coderabbit.yaml @@ -64,20 +64,6 @@ reviews: - Error responses must use structured JSON `{ error: "..." }` or `{ errors: [...] }`, never plain text. - The `set_sismo` before_action pattern (find_by + manual 404) is the project standard; keep it consistent. - # ── Rake Tasks / Background Data Ingestion ────────────────────────── - - path: "lib/tasks/**/*.rake" - instructions: | - The Rake task `sismo:fetch_data` fetches ~10,000 earthquake records from the USGS monthly GeoJSON feed - via HTTParty and bulk-inserts them into PostgreSQL. This task runs as a one-off cron/manual invocation, not as a queued job. - Review checklist: - - HTTP resilience: any new or modified external HTTP call MUST handle timeouts (HTTParty default is none), - network errors (Net::OpenTimeout, Net::ReadTimeout, SocketError), and unexpected/malformed JSON. - - Deduplication: existing logic uses `Sismo.find_by(title:)` — if the dedup key changes, verify uniqueness at the DB level too. - - Performance: bulk operations on thousands of records should consider `insert_all` / `upsert_all` or batching, - not individual `save` calls inside `.each`. - - Logging: use `Rails.logger` instead of bare `puts` for production-safe output. - - Never hardcode the USGS URL without allowing ENV override for testing. - # ── Migrations ────────────────────────────────────────────────────── - path: "db/migrate/**/*.rb" instructions: | @@ -127,7 +113,7 @@ reviews: # ── Gemfile ───────────────────────────────────────────────────────── - path: "Gemfile" instructions: | - Key dependencies: rails 7.2, pg, puma, httparty, will_paginate, rack-cors, dotenv-rails. + Key dependencies: rails 7.2, pg, puma, will_paginate, rack-cors, dotenv-rails. Dev/test: minitest, debug, rubocop, brakeman, bundler-audit. Review checklist: - New gems must justify their addition (avoid bloat in an API-only app). diff --git a/Gemfile b/Gemfile index f94317b..a7204e0 100644 --- a/Gemfile +++ b/Gemfile @@ -52,10 +52,6 @@ end gem 'dotenv-rails', '~> 2.1', '>= 2.1.1', groups: %i[development test] -gem 'httparty', '>= 0.24.0' - -gem 'csv' - gem 'will_paginate', '~> 4.0' gem 'rack-cors', '~> 2.0', '>= 2.0.2' diff --git a/Gemfile.lock b/Gemfile.lock index 8f456b6..e07f1cc 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -90,7 +90,6 @@ GEM concurrent-ruby (1.3.8) connection_pool (2.5.5) crass (1.0.7) - csv (3.3.6) date (3.5.1) debug (1.11.1) irb (~> 1.10) @@ -104,10 +103,6 @@ GEM erubi (1.13.1) globalid (1.4.0) activesupport (>= 6.1) - httparty (0.24.2) - csv - mini_mime (>= 1.0.0) - multi_xml (>= 0.5.2) i18n (1.15.2) concurrent-ruby (~> 1.0) io-console (0.8.2) @@ -134,8 +129,6 @@ GEM mini_portile2 (2.8.9) minitest (5.27.0) msgpack (1.8.4) - multi_xml (0.9.1) - bigdecimal (>= 3.1, < 5) net-imap (0.6.6) date net-protocol @@ -281,10 +274,8 @@ DEPENDENCIES brakeman (~> 6) bundler-audit (~> 0.9) connection_pool (~> 2.4) - csv debug dotenv-rails (~> 2.1, >= 2.1.1) - httparty (>= 0.24.0) minitest (~> 5.22) pg (~> 1.1) puma (>= 5.0) diff --git a/README.md b/README.md index 780b1e7..420ba62 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Telurify API -The Rails backend API service for **Telurify**, a platform that collects, processes, and exposes worldwide seismic activity data from the [USGS Earthquake Hazards Program](https://earthquake.usgs.gov/). Events are collected via a background rake task, exposed through a JSON API, and allow users to submit structured "Did You Feel It?" intensity reports. +The Rails backend API service for **Telurify**, a platform that exposes worldwide seismic activity data collected from the [USGS Earthquake Hazards Program](https://earthquake.usgs.gov/). Events are exposed through a JSON API and allow users to submit structured "Did You Feel It?" intensity reports. > **Note:** The frontend application lives in a separate repository (`telurify-web`), built with Astro and React islands. @@ -15,7 +15,6 @@ flowchart LR ReportsController[ReportsController
POST /v1/sismos/:id/reports] SM[Sismo Model] RM[Report Model] - RT[Rake Task
sismo:fetch_data] RA[Rack::Attack
rate limiting] end @@ -24,8 +23,9 @@ flowchart LR Redis[(Redis / Upstash)] end - subgraph External["External"] - USGS[USGS GeoJSON Feed
all_month.geojson] + subgraph External["External Services"] + Ingestion[Telurify Ingestion] + USGS[USGS Earthquake Hazards Program] end RC --> SM @@ -34,8 +34,8 @@ flowchart LR RA -->|counters| Redis SM --> PG RM --> PG - RT -->|fetch & validate| USGS - RT -->|persist| SM + Ingestion -->|fetch & validate| USGS + Ingestion -->|persist| PG ``` **Component responsibilities:** @@ -43,7 +43,7 @@ flowchart LR | Layer | Responsibility | |---|---| | **Backend API** | Serves paginated, filterable seismic events in a JSON:API-style format and accepts structured intensity reports for events. | -| **Rake task** | Pulls the USGS "Past 30 days" GeoJSON feed, validates ranges (magnitude, latitude, longitude), skips duplicates, and persists records. | +| **Ingestion service** | The [Telurify Ingestion service](https://github.com/Euler-B/Telurify-Ingestion) collects, validates, and persists seismic events. | | **Rack::Attack** | Rate-limits all requests by IP (60 req/min) and throttles the reports endpoint specifically (5 req/min) to prevent spam on a public, unauthenticated endpoint. | | **PostgreSQL** | Stores `sismos` (events) and `reports`. | | **Redis (Upstash)** | Backs `rack-attack`'s distributed rate-limit counters in production. | @@ -55,7 +55,7 @@ flowchart LR **Backend** - Ruby 3.4.10 / Rails 7.2.3 (API-only mode) - PostgreSQL 16 -- `httparty` (USGS feed), `will_paginate`, `rack-cors` +- `will_paginate`, `rack-cors` - `rack-attack` + `redis` (rate limiting, backed by Upstash in production) - Linting/security: `rubocop`, `brakeman`, `bundler-audit` @@ -87,16 +87,6 @@ This single command will: > **Note on Redis:** no local Redis is required for development. `rack-attack` falls back to an in-memory store automatically when `REDIS_URL` / `RACK_ATTACK_REDIS_URL` are unset. -### Load seismic data - -Fetch the latest 30 days of events from USGS into the database: - -```bash -docker compose exec backend bin/rails sismo:fetch_data -``` - -The task reports how many records were created, skipped as duplicates, and rejected by validation. - ### Access the apps | App | URL | diff --git a/lib/tasks/fetch_sismo_data.rake b/lib/tasks/fetch_sismo_data.rake deleted file mode 100644 index 4ff655b..0000000 --- a/lib/tasks/fetch_sismo_data.rake +++ /dev/null @@ -1,58 +0,0 @@ -require 'httparty' -require 'json' - -namespace :sismo do - desc 'Fetch and persist sismo data' - task fetch_data: :environment do - url = 'https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson' - response = HTTParty.get(url) - data = JSON.parse(response.body) - - new_records = 0 - duplicate_records = 0 - invalid_records = 0 - - data['features'].each do |feature| - properties = feature['properties'] - geometry = feature['geometry']['coordinates'] - - title = properties['title'] - existing_sismo = Sismo.find_by(title: title) - - if existing_sismo - duplicate_records += 1 - puts "Registro duplicado: #{title}" - else - sismo = Sismo.new( - title: title, - url: properties['url'], - place: properties['place'], - magType: properties['magType'], - mag: properties['mag'], - tsunami: properties['tsunami'], - external_id: feature['id'], - latitude: geometry[1], - longitude: geometry[0] - ) - - if sismo.valid? - if sismo.save - new_records += 1 - else - puts "Error al guardar el registro: #{sismo.errors.full_messages}" - end - elsif sismo.errors[:mag].include?('La magnitud del sismo debe estar entre -1.0 y 10.0') - invalid_records += 1 - else - duplicate_records += 1 - puts "Registro duplicado: #{sismo.title}" - end - end - end - - puts "Nuevos registros guardados: #{new_records}" - puts "Registros duplicados omitidos: #{duplicate_records}" - puts "Registros inválidos omitidos: #{invalid_records}" - puts 'Datos de los sismos obtenido, validados, y persistidos' - end -end