Objects within layer3domains are missing in the `ndcli list containers` command

[TheRealBecks]

The objects within layer3domains are missing in the ndcli list containers with or without the layer3domain <domain> parameter when no prefix is provided.

dzgaaj@vm-jumphost:~$ ndcli list containers layer3domain 6724-swh-pwcrypt-vrf 10.0.16.64/28
layer3domain: 6724-swh-pwcrypt-vrf
10.0.16.64/28 (Container) comment:re-number, vrf swh-pwcrypt-vrf temp p-to-p
  10.0.16.64/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4
  10.0.16.66/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4
  10.0.16.68/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4
  10.0.16.70/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4
  10.0.16.72/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4
  10.0.16.74/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4
  10.0.16.76/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4
  10.0.16.78/31 (Subnet) pool:de-transfer-swh-pwcrypt-vrf_v4

--> 10.0.16.64/28 will be found

dzgaaj@vm-jumphost:~$ ndcli list containers layer3domain 6724-swh-pwcrypt-vrf

-> empty

dzgaaj@vm-jumphost:~$ ndcli list containers | grep -E "layer3domain" -B 5 -A 5
layer3domain: default
0.0.0.0/1 (Container) RIR:iana comment:ITOUDP-5051 - IPv4 Internet Part 1
  0.0.0.0/8 (Container) comment:reserved reverse_dns_profile:internal source:RFC5735
    0.0.0.0/8 (Available)
  1.0.0.0/8 (Available)
  2.0.0.0/12 (Available)
--
    fdc8::/13 (Available)
    fdd0::/12 (Available)
    fde0::/11 (Available)


layer3domain: 6724-global
fd77:294e:66d3::/48 (Container) comment:Strato DC de.ber.rs
  fd77:294e:66d3::/50 (Container) comment:NOC Backbone
    fd77:294e:66d3::/50 (Available)
  fd77:294e:66d3:4000::/50 (Container) comment:Projects and Customers
    fd77:294e:66d3:4000::/56 (Container) comment:local /64 prefixes

-> Only 6724-global has been listed and all other layer3domains are missing

[Gibheer]

Can you run ndcli list containers layer3domain 6724-swh-pwcrypt-vrf -d and ndcli list containers -d?
That will show some debug output containing the parameters sent to the dim middleware.

For the first command, that should return all containers properly, if the layer3domain is sent.
For the second command it is possible that you have a default layer3domain set in your .ndclirc.

[TheRealBecks]

dzgaaj@vm-jumphost:~$ ndcli list containers layer3domain 6724-swh-pwcrypt-vrf -d
DEBUG - Dim server URL: https://<server_name>/dim
DEBUG - Username: <my_username>
DEBUG - dim call: get_username()
DEBUG - time taken: 0.026
DEBUG - dim result: '<my_username>'
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-swh-pwcrypt-vrf'})
DEBUG - time taken: 0.021
DEBUG - dim result: {'containers': [], 'messages': []}

dzgaaj@vm-jumphost:~$ ndcli list containers -d
DEBUG - Dim server URL: https://<server_name>/dim
DEBUG - Username: <my_username>
DEBUG - dim call: get_username()
DEBUG - time taken: 0.022
DEBUG - dim result: '<my_username>'
DEBUG - dim call: layer3domain_list()
DEBUG - time taken: 0.016
DEBUG - dim result: [{'comment': 'the correct name would be 8560-global',
  'name': 'default',
  'properties': {'rd': '8560:1'},
  'type': 'vrf'},
 {'comment': None, 'name': 'DONOTUSE', 'type': 'DONOTUSE'},
 {'comment': 'Strato global table',
  'name': '6724-global',
  'type': 'global-table'},
 {'comment': 'arsys legacy rfc1918 space',
  'name': '8560-arsys-rfc1918',
  'properties': {'rd': '8560:80'},
  'type': 'vrf'},
 {'comment': 'mgmt/management/oobms (vendor dependent) rt 6724:111',
  'name': '6724-mgmt-vrf',
  'properties': {'rd': '6724:111'},
  'type': 'vrf'},
 {'comment': 'HiDrive internal networks rt 6724:112',
  'name': '6724-hidrive-vrf',
  'properties': {'rd': '6724:112'},
  'type': 'vrf'},
 {'comment': 'Shared Webhosting internal networks rt 6724:113',
  'name': '6724-swh-internal-vrf',
  'properties': {'rd': '6724:113'},
  'type': 'vrf'},
 {'comment': 'Shared Webhosting storage internal networks rt 6724:114',
  'name': '6724-swh-store-vrf',
  'properties': {'rd': '6724:114'},
  'type': 'vrf'},
 {'comment': 'Shared Webhosting pwcrypt internal networks rt 6724:115',
  'name': '6724-swh-pwcrypt-vrf',
  'properties': {'rd': '6724:115'},
  'type': 'vrf'},
 {'comment': 'STRATO to IONOS networks rt 6724:116',
  'name': '6724-ionos-private-vrf',
  'properties': {'rd': '6724:116'},
  'type': 'vrf'},
 {'comment': 'mssd backup networks rt 6724:117',
  'name': '6724-mssd-backup-vrf',
  'properties': {'rd': '6724:117'},
  'type': 'vrf'},
 {'comment': 'mssd sn2g networks rt 6724:118',
  'name': '6724-mssd-sn2g-vrf',
  'properties': {'rd': '6724:118'},
  'type': 'vrf'},
 {'comment': None, 'name': 'nw-services-prelive', 'type': 'rd:8560:999'},
 {'comment': 'test-ionos-cloud',
  'name': 'ICNetwork',
  'properties': {'rd': '54548:123'},
  'type': 'vrf'}]
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': 'default'})
DEBUG - time taken: 7.500
DEBUG - dim result: {'containers': [{'attributes': {'RIR': 'iana',
[...]
    fd77:294e:66d3:6000::/51 (Available)
  fd77:294e:66d3:8000::/49 (Available)

Additionally I searched for layer3domain in my output-file:

zgaaj@vm-jumphost:~$ grep -A 5 -B 5 layer3domain temp.txt
DEBUG - Dim server URL: https://<server_name>/dim
DEBUG - Username: <my_username>
DEBUG - dim call: get_username()
DEBUG - time taken: 0.022
DEBUG - dim result: '<my_username>'
DEBUG - dim call: layer3domain_list()
DEBUG - time taken: 0.016
DEBUG - dim result: [{'comment': 'the correct name would be 8560-global',
  'name': 'default',
  'properties': {'rd': '8560:1'},
  'type': 'vrf'},
--
 {'comment': None, 'name': 'nw-services-prelive', 'type': 'rd:8560:999'},
 {'comment': 'test-ionos-cloud',
  'name': 'ICNetwork',
  'properties': {'rd': '54548:123'},
  'type': 'vrf'}]
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': 'default'})
DEBUG - time taken: 7.500
DEBUG - dim result: {'containers': [{'attributes': {'RIR': 'iana',
                                'comment': 'ITOUDP-5051 - IPv4 Internet Part 1',
                                'reverse_dns_profile': 'public-rev'},
                 'children': [{'attributes': {'comment': 'reserved',
--
                               'ip': 'fd00::/8',
                               'status': 'Container'}],
                 'ip': 'fc00::/7',
                 'status': 'Container'}],
 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': 'DONOTUSE'})
DEBUG - time taken: 0.027
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-global'})
DEBUG - time taken: 0.273
DEBUG - dim result: {'containers': [{'attributes': {},
                 'children': [{'ip': '10.0.0.0/20', 'status': 'Available'},
                              {'ip': '10.0.16.0/23', 'status': 'Available'},
                              {'attributes': {'comment': "'sne ldp for mpls'"},
--
                              {'ip': 'fd77:294e:66d3:8000::/49',
                               'status': 'Available'}],
                 'ip': 'fd77:294e:66d3::/48',
                 'status': 'Container'}],
 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '8560-arsys-rfc1918'})
DEBUG - time taken: 0.022
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-mgmt-vrf'})
DEBUG - time taken: 0.018
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-hidrive-vrf'})
DEBUG - time taken: 0.020
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-swh-internal-vrf'})
DEBUG - time taken: 0.020
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-swh-store-vrf'})
DEBUG - time taken: 0.021
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-swh-pwcrypt-vrf'})
DEBUG - time taken: 0.021
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-ionos-private-vrf'})
DEBUG - time taken: 0.021
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-mssd-backup-vrf'})
DEBUG - time taken: 0.019
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': '6724-mssd-sn2g-vrf'})
DEBUG - time taken: 0.020
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': 'nw-services-prelive'})
DEBUG - time taken: 0.020
DEBUG - dim result: {'containers': [], 'messages': []}
DEBUG - dim call: container_list({'include_messages': True, 'layer3domain': 'ICNetwork'})
DEBUG - time taken: 0.021
DEBUG - dim result: {'containers': [], 'messages': []}
layer3domain: default
0.0.0.0/1 (Container) RIR:iana comment:ITOUDP-5051 - IPv4 Internet Part 1 reverse_dns_profile:public-rev
  0.0.0.0/8 (Container) comment:reserved reverse_dns_profile:internal source:RFC5735
    0.0.0.0/8 (Available)
  1.0.0.0/8 (Available)
  2.0.0.0/12 (Available)
--
    fdc8::/13 (Available)
    fdd0::/12 (Available)
    fde0::/11 (Available)


layer3domain: 6724-global
10.0.0.0/8 (Container)
  10.0.0.0/20 (Available)
  10.0.16.0/23 (Available)
  10.0.18.0/23 (Container) comment:'sne ldp for mpls'
    10.0.18.0/23 (Available)

All non-default-layer3domains have an empty result.

That's my config:

dzgaaj@vm-jumphost:~$ cat .ndclirc
server=https://<server_name>/dim
username=<my_username>

[Gibheer]

Okay, something weird is going on. I have looked through the code and it seems okay.
There are even testcases that show, that it must have worked at least in the past: https://github.com/1and1/dim/blob/master/dim-testsuite/t/container-layer3domain.t

I will try to reproduce the issue, so that I can come up with a fix. May take some time though.

[zeromind]

Seems like DIM currently does this:
https://github.com/1and1/dim/blob/dim-5.0.2/dim/dim/rpc.py#L857-L867
When one does not provide a container/prefix, it looks for an Ipblock that does not have a parent (parent_id=None), and is of type Container.
It's trying to find the top-most container to return the "whole" tree for the given layer3domain.

However, in the layer3domain from this issue, there are only Ipblocks with a parent.

>> l3d.name
'6724-swh-pwcrypt-vrf'
>>> dim.models.Ipblock.query.filter_by(parent_id=None, layer3domain=l3d).all()
[]

parent_ids of Ipblocks in the layer3domain:

>>> [ipb.parent_id for ipb in dim.models.Ipblock.query.filter_by(layer3domain=l3d).all()]
[7991830, 7991831, 7991831, 7727504, 7935530, 7935533, 7935533, 7935530, 7968862, 7968862, 7935530, 7968865, 7968865, 7935530, 7968868, 7968868, 7935530, 7968871, 7968871, 7935530, 7968874, 7968874, 7935530, 7968877, 7968877, 7935530, 7968880, 7968880, 7968967, 7968883, 7968883, 7968968, 7968886, 7968886, 7968968, 7968889, 7968889, 7968968, 7968892, 7968892, 7968968, 7968895, 7968895, 7968969, 7968898, 7968898, 7968969, 7968901, 7968901, 7968969, 7968904, 7968904, 7968969, 7968907, 7968907, 7968969, 7968910, 7968910, 7968969, 7968913, 7968913, 7968969, 7968916, 7968916, 7968969, 7968919, 7968919, 7727506, 7968922, 7968923, 7968924, 7968924, 7968923, 7968927, 7968927, 7968923, 7968930, 7968930, 7968922, 7968933, 7968934, 7968934, 7968933, 7968937, 7968937, 7968933, 7968940, 7968940, 7727506, 7968943, 7968944, 7968945, 7968945, 7968944, 7968948, 7968948, 7968944, 7968951, 7968951, 7968943, 7968954, 7968955, 7968955, 7968954, 7968958, 7968958, 7968954, 7968961, 7968961, 7727505, 7968967, 7968967]

Not sure what went wrong there, I think this should not happen.
Will have to dig deeper.