Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 321899 - Inaccurate / missing data in CSV output
Summary: Inaccurate / missing data in CSV output
Status: CLOSED FIXED
Alias: None
Product: STEM
Classification: Technology
Component: Simulation (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows XP
: P1 normal (vote)
Target Milestone: 1.1.2   Edit
Assignee: Stefan Edlund CLA
QA Contact: James Kaufman CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-08-05 12:11 EDT by Greg CLA
Modified: 2011-07-18 14:59 EDT (History)
1 user (show)

See Also:


Attachments
Demo scenario which generated bug (10.14 KB, application/zip)
2010-08-05 12:12 EDT, Greg CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Greg CLA 2010-08-05 12:11:29 EDT
- For Level 0 and 1 log files, there is no output value every other day (i.e. 0 values for every other iteration). This was noticed in ‘E’, ‘I’, and ‘Incidence’ data. Not an evident problem in ‘S’ and ‘R’. 

- Data in Level 2 does not add up to data in Level 1

- The labels for Level 0 and 1 data include all the path information – e.g. stem://org.eclipse.stem/node/transport/pipe/US-PA. This doesn’t occur in Level 2 data

- The difference between the initial total of S,E,I, and R and the final SEIR total are not equal to the total death count, as would be expected
Comment 1 Greg CLA 2010-08-05 12:12:26 EDT
Created attachment 175956 [details]
Demo scenario which generated bug
Comment 2 James Kaufman CLA 2010-08-05 12:57:45 EDT
Gregoray,
Stefan and I have started to look at this. The bug is in the air transportation but we wanted to document some initial observations.

1) The Level 0 and Level 1 data being logged is only data corresponding to the air transportation system. We do not log the state level and country level data (it should just be the aggregate - but see (2) below.
2) The question of deaths not adding up may be due to the fact that when using air transportation, people are actually moved from counties to the air transport nodes and they need to be counted. We will look at this.
3) You only see pipe nodes at level 0 and 1 (and not at level 2) because the first (lowest level) air transport nodes are at Level 1 (state level) and the counties connect to them. This is NOT a bug but we do need to document air transport..
4) We were able to reproduce the fact that data is logged every other cycle for E, I. This is only happening to the air transport nodes (the county nodes all look good). 
5) We don't see you last observation (The difference between the initial total of S,E,I, and R and the final SEIR total are not equal to the total death count, as would be expected) because this scenario has no death rate. Can you please send the scenario where that occurs.

We will continue to work on this today and should have more news for you soon.
Comment 3 James Kaufman CLA 2010-08-05 13:06:04 EDT
Changed bug status to Priority 1. User bug.
Comment 4 Stefan Edlund CLA 2010-08-11 21:57:35 EDT
We have fixed the most serious issue related to this bug, the fact that disease deaths were not accounted for so the population numbers did not add up as expected. The solution was fairly deep down in the STEM solver code, if someone wants more detail I'd be happy to provide it. It now works as expected for models containing both air transportation and migration. IMPORTANT: Disease Deaths.csv is now cumulative, each row contains the total disease deaths from the start of the simulation.

We still keep the log files for the level 0 and 1 (when running a level 2 scenario). Those log files contains data on air transportation systems that does not have any real use. We will decide whether to remove those from the log later.

In fixing this bug, I also improved the log for population models. Births.csv is the cumulative births (from background population birth rate) and Deaths.csv is the cumulative deaths (from background population death rate as well as disease deaths).

Another important thing to keep in mind (we mentioned it on the STEM call): Use the Runge Kutta Integration solver if you want to do real science using STEM. The Finite Difference solver (while it runs fast), is mostly just for demos. I verified and tested this bug using the integrating solver.
Comment 5 Stefan Edlund CLA 2010-08-23 17:24:26 EDT
Fixed
Comment 6 James Kaufman CLA 2011-07-18 14:59:08 EDT
Fixed