›

Layer Operations using TilingProcessor?

ManzAutomationsManzAutomations
in Python scripting

Hi there, I am hoping for some guidance. I got this error while adding two regions together:

Here is my code (In this case, I am just adding two layers. I have other functions to subtract, and, not, xor, etc):

    def add_layer(layout, cell, firstlayer, otherlayers, layer_number):
        # Add other layers to the first layer
        new_region = pya.Region(cell.begin_shapes_rec(firstlayer))
        for layer_index in otherlayers:
            region2 = pya.Region(cell.begin_shapes_rec(layer_index))
            new_region += region2
        new_layer = create_layer(layout, layer_number)
        cell.shapes(new_layer).insert(new_region)

        return new_layer

Basically, it seems like the Tiling Processor is the way to go here. There are some scenarios in which the regions are way too large. So I am curious what it would look like to do the same operation but with tiling instead in order to decrease the amount of memory that the operation takes to do the job. I could not find sufficient information regarding an example in this case.

If anybody has any ideas on how to make this function better in regard to functionality, feel free to give any advice. I am still very new to this API.

Comments

  • RawrRangerRawrRanger
    edited July 2023

    Hi ManzAutomations

    Tile processer can take multiple layers as parameters through input and output,
    and the operation for those layers is defined during tp.queue is being called.

    short example

    tp  = pya.TilingProcessor()
tp.input("in_0", cell.begin_shapes_rec(layout.layer(0, 0))) 
tp.input("in_1", cell.begin_shapes_rec(layout.layer(1, 0)))
tp.output("out", layout, cellID, layout.layer(3,0))
tp.queue("_output(out, in_0 - in_1)")
tp.tile_size(50.0, 50.0)
tp.execute("tiled operation")

    a full example
    A full example can be run using the attached gds file

    import pya
mainWindow    = pya.Application.instance().main_window()
layoutView    = mainWindow.current_view()  
cellView      = layoutView.active_cellview()
layout        = cellView.layout()

def tile_process_layer(layout, cell_name, inlayers, outLayer, operation):
    tp     = pya.TilingProcessor()
    cell   = layout.cell(cell_name)
    cellID = cell.cell_index()

    for i, l_dt in enumerate(inlayers):
        tp.input(f"in_{i}", cell.begin_shapes_rec(layout.layer(*l_dt)))

    tp.output("out", layout, cellID, layout.layer(*outLayer))

    inOprStr = operation.join(["in_%d" % (i) for i in range(len(inlayers))])
    queueStr = f"_output(out, {inOprStr})"
    tp.queue(queueStr)
    tp.tile_size(50.0, 50.0)
    tp.execute(f"tiled {operation}")

def add_layer(layout, cell_name, inlayers, outLayer):
    tile_process_layer(layout, cell_name, inlayers, outLayer, "+")

def subtract_layer(layout, cell_name, inlayers, outLayer):
    tile_process_layer(layout, cell_name, inlayers, outLayer, "-")

add_layer(     layout, "TOP", [(1, 0), (2, 0), (3, 0), (4, 0)],         (5, 0))
#queue string = _output(out, in_0 + in_1 + in_2 + in_3)

subtract_layer(layout, "TOP", [(0, 0), (1, 0), (2, 0), (3, 0), (4, 0)], (6, 0))
#queue string = _output(out, in_0 - in_1 - in_2 - in_3 - in_4)
layoutView.add_missing_layers()
  • MatthiasMatthias

    @RawrRanger Thanks for sharing this code!

    I would like to add that the tiling processor only solves the issue of temporary memory allocation. That means that for example for boolean operations, the memory overhead is reduced because intermediate data (in that case the edge representation of the polygons) is only created for the tile.

    The tiling process does not solve the problem of storage needed to keep the results. In the case of the "+" operation, it will not help and there is no internal stored needed - when you run out of memory, it's simply the result of that operation that takes a lot of memory as it will implicitly flatten the layout.

    The tiling processor can help to reduce memory footprint, if you push your whole process into the "queue" of the tiling processing. So all involved operations up the "output" statement. If you do that properly, intermediate results will only be computed for individual tiles which means less memory. Still the result of the tiling processor (the final product) will be flat. That itself can become an issue. In addition, not all operations are suitable for being put into a tiling processor's queue.

    There is no simple solution. Memory and performance optimization techniques heavily depend on the layout style of your input files. There is no silver bullet.

    Usually maintaining the hierarchy helps. Simply adding two layers can be done cell wise (Layout#copy_layer does this in case you stay inside the same layout object). Deep mode may be an option. The key to this feature is the DeepShapeStore class (https://www.klayout.de/doc-qt5/code/class_DeepShapeStore.html#k_1).

    If hierarchy is not available you can try to optimize memory - for example making sure you do not keep Region objects longer than needed (Python uses reference counting, so make sure there are no object references). You can force a memory release by using "Region#_destroy". Also in-place operations are often more efficient.

    There are still cases however, when you simply need more memory.

    Matthias

Sign In or Register to comment.
Powered by Vanilla