how to partially write out oas?

dion

Hi,
I have to build a large oas file from lots of small files.

For an example, read in geometry from one small oas, then do some polygon operations,
then save data as new cell in final oas.

Now, I have to read all small files to memory and then write them out.
It takes about 70~80G memory. (There are about 2000 small oas)

Could I partially write out cell data after one small oas processed?
Then writing out cell reference information in the end.
It could reduce memory usage.

Thanks,
Dion

Matthias

Hi Dion,

this may be possible in some flavours of OASIS, but not in all of them: some OASIS flavours require tables to be built upfront, so you need to know the full file content before you can write it. One can overcome this by a two-pass approach and modify the file's header afterwards.

Nevertheless, KLayout OASIS write is single-pass and not optimized for this use case. In order to get the implementation you look for, you'd have to modify the OASIS writer code in a very specific way.

But maybe there are some other ways to optimize memory:

• If you script the generation code, make sure the big objects (specifically Layout) are deleted
  as soon as they are no longer required (use object._destroy
• Construct Layout objects as "non-editable" (use "Layout(false)"). Some operations are not supported
  in this mode, but most are. It's worth a try.
• Do the operation first on all files while writing temporary files (2000 read + 2000 OASIS writes). Then
  read again and combine the layouts in non-editable mode. Advantage: the OASIS write will compress
  the shapes and when using non-editable mode, the compression is retained.

Kind regards,

Matthias

dion

Thanks for your suggestion :)
I will try it.

Thanks,
Dion

dion

Hi Matthias,
I tried your suggestion,

Do the operation first on all files while writing temporary files (2000 read + 2000 OASIS writes). Then read again and combine the layouts in non-editable mode.

For step 2, I open layout in editable mode, then open 2000 oasis one by one in non-editable mode, then using 'copy_shapes' in cell class to combine final output.

Is it correct? It seems not reduce memory usage.

Thanks,
Dion