This is a snapshot of Indico's old Trac site. Any information contained herein is most probably outdated. Access our new GitHub site here.

Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#983 closed defect (fixed)

maintenance/cleanupWebSessions.py does not scale

Reported by: makub Owned by: jatrzask
Priority: critical Milestone: v0.99.0
Component: General Version: 0.97.0
Keywords: Cc:

Description

The script maintenance/cleanupWebSessions.py does not work for millions of web sessions, because it first collects all sessions keys, and then removes them. The script exhausts memory on a machine with 4GB memory.

The solution is to simply find and remove sessions in batches of small sizes, like 1000.

Attachments (3)

cleanupWebSessions.py (1.1 KB) - added by makub 4 years ago.
a script that deletes in batches
ncar-super-computers-web.jpg (17.8 KB) - added by Slavon 3 years ago.
.
piwik.php (2.8 KB) - added by Slavon 2 years ago.
Information about Neti Pot http://netipotby.com

Download all attachments as: .zip

Change History (10)

Changed 4 years ago by makub

a script that deletes in batches

comment:1 Changed 4 years ago by jbenito

  • Milestone set to v0.98.1

comment:2 Changed 3 years ago by jbenito

  • Milestone changed from v0.98.1 to v0.99.0
  • Owner set to jatrzask
  • Priority changed from normal to critical
  • Status changed from new to assigned

comment:3 Changed 3 years ago by jatrzask

  • Status changed from assigned to in_work

comment:4 Changed 3 years ago by jatrzask

  • Status changed from in_work to awaiting_merge

comment:5 Changed 3 years ago by pferreir

  • Status changed from awaiting_merge to merging

Martin,

Thanks for the contribution, but I believe there is a better solution which involves using iterators.
I will add the fix, and take the opportunity to finally move this to a scheduler task.

Give it a try and let us know, please.

comment:6 Changed 3 years ago by Jakub Trzaskoma <jakub.piotr.trzaskoma@…>

  • Resolution set to fixed
  • Status changed from merging to closed

In e9e2cbf8c04a645857857cd4c39f10b046e98bb0/indico:

[FIX] Cleanup web sessions script is not scalable

comment:7 Changed 3 years ago by pferreir

fe5e148046b7acba3cb92ea1f09a107a3f622fc9 contains the solution using iterators.

Version 0, edited 3 years ago by pferreir (next)

Changed 3 years ago by Slavon

Changed 2 years ago by Slavon

Information about Neti Pot http://netipotby.com

Note: See TracTickets for help on using tickets.