[elixir] fix: processing of ODS loads without snapshot load

[grejdi-mbta]

This PR addresses 2 bugs:

• Stop any queries from running for tables that haven't had any loads come through.
• Prevent the ingestion of loads that don't have a snapshot load. Due to deployment (one container coming online and the other offline), some sort of race condition has occurred where some tables lost out on their snapshot loads being inserted. I believe it has to do with this filtering logic. I haven't been able to reproduce locally, but this bug fix addresses this bad state, which can also occur by Cubic sending over data that we don't understand, such as an update in the name of the snapshot load file. Currently, CT loads are going to the error bucket because they don't have they don't have have destination, due to the missing snapshot.

Lastly, I'm planning on addressing the filtering logic of the new load objects to have it be more inclusive, and look back further (maybe an hour). This will be in another PR.

[github-actions]

Coverage of commit a80af5e

Summary coverage rate:
  lines......: 93.5% (476 of 509 lines)
  functions..: 73.7% (171 of 232 functions)
  branches...: no data found

Files changed coverage rate:
                                                           |Lines       |Functions  |Branches    
  Filename                                                 |Rate     Num|Rate    Num|Rate     Num
  ===============================================================================================
  lib/ex_cubic_ingestion/schema/cubic_load.ex              | 100%     49| 100%    20|    -      0

Download coverage report

[github-actions]

Coverage of commit 28c2a9e

Summary coverage rate:
  lines......: 93.5% (476 of 509 lines)
  functions..: 73.7% (171 of 232 functions)
  branches...: no data found

Files changed coverage rate:
                                                           |Lines       |Functions  |Branches    
  Filename                                                 |Rate     Num|Rate    Num|Rate     Num
  ===============================================================================================
  lib/ex_cubic_ingestion/schema/cubic_load.ex              | 100%     49| 100%    20|    -      0

Download coverage report

[paulswartz]

We both had concerns about this change in #77. Given that it's experiencing issues that we can't reproduce in testing, should we consider reverting the change and trying a different approach to stabilize the container?

[grejdi-mbta]

We both had concerns about this change in #77. Given that it's experiencing issues that we can't reproduce in testing, should we consider reverting the change and trying a different approach to stabilize the container?

With the latest Qlik restart, everything has been working flawlessly. I have a theory on how to reproduce, just need to stage a little more.
Regarding the reverting, I rather not due to the performance issues. I'd like to do something like this: #84 to capture anything that might have been left behind in the last 24 hours. Still thinking through all the implications.

[github-actions]

Coverage of commit 8e06081

Summary coverage rate:
  lines......: 93.5% (476 of 509 lines)
  functions..: 73.7% (171 of 232 functions)
  branches...: no data found

Files changed coverage rate:
                                                           |Lines       |Functions  |Branches    
  Filename                                                 |Rate     Num|Rate    Num|Rate     Num
  ===============================================================================================
  lib/ex_cubic_ingestion/schema/cubic_load.ex              | 100%     49| 100%    20|    -      0

Download coverage report

[paulswartz]

Were you able to reproduce the issue locally?

[grejdi-mbta]

Were you able to reproduce the issue locally?

No. I had some theories that didn't work out. I have one remaining, but it will involve running two instances of the app at once and a few hundred data files, which would create the conditions during a deployment. I'm hoping to avoid this scenario in general, since the design of the app doesn't take into account multiple instances. I'll be implementing some changes to the ECS container to stop the container before starting another, so that should address any issues stemming from or during a deployment.

[paulswartz]

If you can't reproduce the issue, and everything is working with the smaller Qlik batches, do you still need to make this change?

[grejdi-mbta]

If you can't reproduce the issue, and everything is working with the smaller Qlik batches, do you still need to make this change?

This code protects the integrity of the process further, so I'd like to deploy it.

[paulswartz]

🍰