Multi-step OASIS stream file writer

Comments

• Matthias

  July 2024 edited July 2024

  Do you have an application that really needs that?

  Basically, such a thing is scary. Stream writers, in particular OASIS, benefit a lot from having all the knowledge about the layout and hierarchy at hand. For example, if cell2 is a child cell of cell1 - how can you compute the bounding box of cell1 then?

  If you really need to write huge layouts and come across a memory barrier, you can try writing each cell into separate OASIS. This benefits from the OASIS compression - specifically shape array formation.
  In another step you combine these individual layouts. If you load them into a non-editable layout, it will keep the OASIS compression, hence require a lot less memory.

  But I have been creating huge cells already myself without seeing the need to do the multi-step thing.

  Here is an experiment:

  import klayout.db as kdb
  
  ly = kdb.Layout()
  
  l1 = ly.layer(1, 0)
  c1 = ly.create_cell("TOP")
  shapes = c1.shapes(l1)
  
  for i in range(0, 10000):
    for j in range (0, 10000):
      shapes.insert(kdb.Box(i * 10, j * 10, i * 10 + 9, j * 10 + 9))
  
  ly.write("test.oas")
  

  This script creates 100M of boxes. It runs about 5 minutes (C++: much, much faster) and consumes 1.6G memory (2.2G virtual) on Ubuntu 22. That is consistent with 16 bytes per box. During OASIS write, memory peaks at 3.2G because in that extreme single-cell and single-layer case, the OASIS writer produces a flat map for repetition extraction. Finally, the OASIS file is only 400 Bytes.

  Matthias
• ryanke

  July 15

  Hi @Matthias,

  Could you elaborate more about what you meant by “C++: much, much faster.” Were you referring to the performance difference when running the process in C++ instead of using a Python script?

  I’m currently looking into ways to speed up the OAS write process, so I’d really appreciate any suggestions or insights you might have.

  Best regards,
  Ryan
• Matthias

  July 16

  Hi @Ryan,

  yes, I meant coding the generation of the shapes in C++ is much faster than in Python. I was referring to that.

  Writing does not benefit from that, as the OASIS writer is already coded in C++.

  Matthias
• dion

  August 27

  I added a patch to dbOASISWriter.cc to implement a write-step feature. For example, when write step = 2, it skips the OAS file header and outputs only specified cells, writing their data into byte arrays. This step can be performed in parallel using multiple CPU cores. In Python, multiple byte arrays are generated, which are then combined into a single bytes cache that records the byte positions of each cell. Then, when write step = 4 is used, it reads the byte positions of all cells from the cache and generates the complete OASIS file. When reaching a specific cell during output, it directly retrieves and writes the content from the bytes cache, significantly accelerating the overall output speed.

  p1.txt 17.6K