Per-Container widget show "No Data Availalbe"

[sjtuhjh]

The basic widget(such as CPU, network, memory) work fine, but all widget related with container show "No Data Available".

For example, the following case tries to show both "Memory" and "Per-Container CPU Utilization" whose request and response are listed as follows. However, only "Memory" display properly.

Http Request:
http://192.168.1.86:44323/pmapi/1107686313/_fetch?names=containers.cgroup,containers.name,cgroup.cpuacct.usage,mem.util.free

Http Resp:

{"timestamp":{"s":1501643762,"us":351117 }, "values":[{"pmid":4194310,"name":"containers.cgroup","instances":[
{"instance":0, "value":"/system.slice/docker-8b064b8d4cbc286754fcd1031d9a9008dfa5be8adc710e3fb2896cc1cd06656f.scope" },{"instance":1, "value":"/system.slice/docker-72b67e4d6de18a493e9beb3b42dafd0680df2391e3a8855e37ab83919b516f19.scope" },{"instance":2, "value":"/system.slice/docker-6b5047d5173da546e77691ea96195e7c98876bf721ec162c7fb8be9375987a5b.scope" },{"instance":3, "value":"/system.slice/docker-6f1577377ed85472fda4518b83bdd2487276cf3eae482d5b8c1f9605f6933859.scope" },{"instance":4, "value":"/system.slice/docker-f0d96149d6d4d71dec401f055340307e1b2fb4bd6a4e8453fc19e11200930355.scope" },{"instance":5, "value":"/system.slice/docker-add03b69d812c97dd227994213a9b84627b301be56ed5e468d2925360806ac0e.scope" },{"instance":6, "value":"/system.slice/docker-c4ffe01954d4c1048ec2084d97805329d6a167a900ea4743e37e78327dca725d.scope" },{"instance":7, "value":"/system.slice/docker-762d2a3cd44d40319e8cf0821fb4dc672b82864c2569de4f03e630509a34201b.scope" },{"instance":8, "value":"/system.slice/docker-e4e7d332abc5526096d7b6553f09b429481bae8def1bc5053109f1932a6aa06f.scope" },{"instance":9, "value":"/system.slice/docker-4ae462c82e03062ac8c1d06a37652033951828734a96244309450b2369dc482b.scope" },{"instance":10, "value":"/system.slice/docker-640839fd833fee1bbaef3fa85b65cc7460f38ad32c5e8216b54b034d5c09615f.scope" },{"instance":11, "value":"/system.slice/docker-2751350dfd3109734b1b76b9a516d1461459e788a3fb6c6151868311c4029b21.scope" },{"instance":12, "value":"/system.slice/docker-d8a84a531c7cca0386b7b44378c0f1556346aea369d00d26ef8c37120367b05a.scope" },{"instance":13, "value":"/system.slice/docker-01c41aecc1b3724eacfede02167ae046ed00162c4d0aba78894b0375637e73f8.scope" },{"instance":14, "value":"/system.slice/docker-bdd6dfbec46824052f85322898b6a776add3f12ac2d49e340d0ed68ef7f0a83a.scope" },{"instance":15, "value":"/system.slice/docker-d3bc311ea55e7f648613f75a5b0a6a4b15750cea8b0c4c1979481f67ce565706.scope" },{"instance":16, "value":"/system.slice/docker-5e6ed946fa87953709d607bc20af2b74988ed68a53f8e46a17c190682fa1fc10.scope" },{"instance":17, "value":"/system.slice/docker-16c1256c88b9ae634093376c93ede7ea1f7c0dcb0c4173f9b7044e2ac141220a.scope" },{"instance":18, "value":"/system.slice/docker-bd33ba79a1500a2e82a13401006cf158711d7ba6ec73bf3aaa0e20fef959776b.scope" },{"instance":19, "value":"/system.slice/docker-b7af500211692e5a72bfa639ec4b1486036f61b38bba2af11d545339e0071a09.scope" },{"instance":20, "value":"/system.slice/docker-bf0b7c905684ae4ea70264dc9478da3aa4957e4ddce3081cd4abb7e01889f112.scope" },{"instance":21, "value":"/system.slice/docker-0b532d67d98c8b052c482915877e6f528a89c6065f16e71095b60ab1d24f857e.scope" },{"instance":22, "value":"/system.slice/docker-e0d43f796936c746d85337e2b0e2de59d8e1252f0955bd2894d8826ad58e4372.scope" },{"instance":23, "value":"/system.slice/docker-9774384973a65e53ef91fddbcecf5b79099936b98b15aa79555584a6bfc28925.scope" },{"instance":24, "value":"/system.slice/docker-c541796508d90c431445ef52b5f7803f56d5f6520bc3bf5ab757fc26cbceaa52.scope" }]},
{"pmid":4194305,"name":"containers.name","instances":[
{"instance":0, "value":"k8s_POD_monitoring-grafana-4072518999-5fdd8_kube-system_fe7a044e-6833-11e7-a978-c0a802ee0004_0" },{"instance":1, "value":"k8s_POD_kubernetes-dashboard-3981181111-v5t64_kube-system_af9c83c0-6873-11e7-a978-c0a802ee0004_0" },{"instance":2, "value":"k8s_POD_kube-scheduler-centos_kube-system_f24680dde0a73fe24c34094896acb898_0" },{"instance":3, "value":"k8s_kube-scheduler_kube-scheduler-centos_kube-system_f24680dde0a73fe24c34094896acb898_0" },{"instance":4, "value":"k8s_kube-flannel_kube-flannel-ds-8v27n_kube-system_2f4b65ce-6832-11e7-a978-c0a802ee0004_0" },{"instance":5, "value":"k8s_grafana_monitoring-grafana-4072518999-5fdd8_kube-system_fe7a044e-6833-11e7-a978-c0a802ee0004_0" },{"instance":6, "value":"k8s_kubernetes-dashboard_kubernetes-dashboard-3981181111-v5t64_kube-system_af9c83c0-6873-11e7-a978-c0a802ee0004_0" },{"instance":7, "value":"k8s_install-cni_kube-flannel-ds-8v27n_kube-system_2f4b65ce-6832-11e7-a978-c0a802ee0004_0" },{"instance":8, "value":"k8s_kubedns_kube-dns-2286869516-0hz5f_kube-system_17607aee-6832-11e7-a978-c0a802ee0004_0" },{"instance":9, "value":"k8s_heapster_heapster-2843423903-b3dgm_kube-system_251b9848-6833-11e7-a978-c0a802ee0004_0" },{"instance":10, "value":"k8s_POD_kube-apiserver-centos_kube-system_c5d63b165885460721c08e062c533ea5_0" },{"instance":11, "value":"k8s_POD_etcd-centos_kube-system_b4c88f8c66b21d09a76829f5730c0b2b_0" },{"instance":12, "value":"k8s_POD_kube-proxy-4fkls_kube-system_1751dc50-6832-11e7-a978-c0a802ee0004_0" },{"instance":13, "value":"k8s_sidecar_kube-dns-2286869516-0hz5f_kube-system_17607aee-6832-11e7-a978-c0a802ee0004_0" },{"instance":14, "value":"k8s_kube-proxy_kube-proxy-4fkls_kube-system_1751dc50-6832-11e7-a978-c0a802ee0004_0" },{"instance":15, "value":"k8s_etcd_etcd-centos_kube-system_b4c88f8c66b21d09a76829f5730c0b2b_0" },{"instance":16, "value":"k8s_POD_heapster-2843423903-b3dgm_kube-system_251b9848-6833-11e7-a978-c0a802ee0004_0" },{"instance":17, "value":"k8s_kube-controller-manager_kube-controller-manager-centos_kube-system_16bd019c32d4cbbbdf6516aba0701adf_0" },{"instance":18, "value":"k8s_POD_kube-controller-manager-centos_kube-system_16bd019c32d4cbbbdf6516aba0701adf_0" },{"instance":19, "value":"k8s_POD_kube-flannel-ds-8v27n_kube-system_2f4b65ce-6832-11e7-a978-c0a802ee0004_0" },{"instance":20, "value":"k8s_POD_monitoring-influxdb-562486248-3h3dx_kube-system_2550ef2a-6833-11e7-a978-c0a802ee0004_0" },{"instance":21, "value":"k8s_dnsmasq_kube-dns-2286869516-0hz5f_kube-system_17607aee-6832-11e7-a978-c0a802ee0004_0" },{"instance":22, "value":"k8s_influxdb_monitoring-influxdb-562486248-3h3dx_kube-system_2550ef2a-6833-11e7-a978-c0a802ee0004_0" },{"instance":23, "value":"k8s_kube-apiserver_kube-apiserver-centos_kube-system_c5d63b165885460721c08e062c533ea5_0" },{"instance":24, "value":"k8s_POD_kube-dns-2286869516-0hz5f_kube-system_17607aee-6832-11e7-a978-c0a802ee0004_0" }]},
{"pmid":12624898,"name":"cgroup.cpuacct.usage","instances":[
{"instance":0, "value":4933172881291360 },{"instance":1, "value":129896343034780 },{"instance":2, "value":34344311615640 },{"instance":3, "value":714945122980 },{"instance":4, "value":714918888420 },{"instance":5, "value":26234560 },{"instance":6, "value":796197274900 },{"instance":7, "value":796167501180 },{"instance":8, "value":29773720 },{"instance":9, "value":17739949351600 },{"instance":10, "value":22744280 },{"instance":11, "value":16780276211800 },{"instance":12, "value":7928199924620 },{"instance":13, "value":7928177683920 },{"instance":14, "value":22240700 },{"instance":15, "value":15779240 },{"instance":16, "value":983657548940 },{"instance":17, "value":25466600 },{"instance":18, "value":983632082340 },{"instance":19, "value":371512625340 },{"instance":20, "value":756403400 },{"instance":21, "value":370734284620 },{"instance":22, "value":21937320 },{"instance":23, "value":5497990940440 },{"instance":24, "value":24815180 },{"instance":25, "value":5497966125260 },{"instance":26, "value":95552031482520 },{"instance":27, "value":3357321921020 },{"instance":28, "value":19876880 },{"instance":29, "value":3355518204280 },{"instance":30, "value":31112849747960 },{"instance":31, "value":31112829336200 },{"instance":32, "value":20411760 },{"instance":33, "value":6314037052680 },{"instance":34, "value":4344877118660 },{"instance":35, "value":341030693080 },{"instance":36, "value":26561020 },{"instance":37, "value":1628102679920 },{"instance":38, "value":51031145037860 },{"instance":39, "value":20154260 },{"instance":40, "value":51031124883600 },{"instance":41, "value":1484065684220 },{"instance":42, "value":841638212120 },{"instance":44, "value":59236030660 },{"instance":45, "value":4789348258625900 },{"instance":46, "value":706748098020 },{"instance":47, "value":1563100 },{"instance":48, "value":0 },{"instance":49, "value":254087712480 },{"instance":50, "value":0 },{"instance":51, "value":0 },{"instance":52, "value":0 },{"instance":53, "value":0 },{"instance":54, "value":0 },{"instance":55, "value":0 },{"instance":56, "value":0 },{"instance":57, "value":0 },{"instance":58, "value":0 },{"instance":59, "value":47681864860 },{"instance":60, "value":374043482500 },{"instance":61, "value":5654771940 },{"instance":62, "value":0 },{"instance":63, "value":819622640 },{"instance":64, "value":0 },{"instance":65, "value":0 },{"instance":66, "value":85684428284820 },{"instance":67, "value":0 },{"instance":68, "value":0 },{"instance":69, "value":4656702398478900 },{"instance":70, "value":0 },{"instance":71, "value":0 },{"instance":72, "value":0 },{"instance":73, "value":0 },{"instance":74, "value":0 },{"instance":75, "value":0 },{"instance":76, "value":0 },{"instance":77, "value":0 },{"instance":78, "value":329761660 },{"instance":79, "value":3396200 },{"instance":80, "value":0 },{"instance":81, "value":0 },{"instance":82, "value":61116673340 },{"instance":83, "value":0 },{"instance":84, "value":0 },{"instance":85, "value":0 },{"instance":86, "value":0 },{"instance":87, "value":23126317060 },{"instance":88, "value":63612599380 },{"instance":89, "value":0 },{"instance":90, "value":0 },{"instance":91, "value":406311588180 },{"instance":92, "value":1870110241300 },{"instance":93, "value":2578555477820 },{"instance":94, "value":0 },{"instance":95, "value":0 },{"instance":96, "value":0 },{"instance":97, "value":16466060 },{"instance":98, "value":9045939800140 },{"instance":99, "value":57367804500 },{"instance":100, "value":0 },{"instance":101, "value":0 },{"instance":102, "value":0 },{"instance":103, "value":0 },{"instance":104, "value":0 },{"instance":105, "value":0 },{"instance":106, "value":0 },{"instance":107, "value":0 },{"instance":108, "value":1573310840 },{"instance":110, "value":81827591800 }]},
{"pmid":251659266,"name":"mem.util.free","instances":[
{"instance":-1, "value":1368844 }]}]}

[spiermar]

That's interesting. Seems like the the values are being returned from PCP, so this means it's likely one of the container id/name checks.
Do you have a specific container selected in drop down?

[spiermar]

Vector checks a few things before adding container data to the widget. First, it check if it has metadata for that container. https://github.com/Netflix/vector/blob/master/src/app/components/containermetadata/containermetadata.service.js#L78

Then it check if a container was selected in the drop down:
https://github.com/Netflix/vector/blob/master/src/app/components/containermetadata/containermetadata.service.js#L187

And if a container filter was applied:
https://github.com/Netflix/vector/blob/master/src/app/components/containermetadata/containermetadata.service.js#L179

Judging by the response, if no filter is applied and no container name is selected, I believe it's a bug dealing with the Kubernetes container Ids. An issue parsing the /system.slice/docker-*.scope format.

[sjtuhjh]

@spiermar, I have tryied both "all" or "specific container" in drop down, but none of them work properly.

[spiermar]

@sjtuhjh I think I know where the problem is, so I'll try to get a fix by the end of the week. Once I do, would you be able to build Vector from master and check it?

[sjtuhjh]

@spiermar Sure! Please just let me know when you have done it.

[sjtuhjh]

@spiermar, have you fixed this issue?

[spiermar]

@sjtuhjh Not yet. Couldn't get the time yet.

[spiermar]

Fixed in 1520b62.

Right now the metric filtering mechanism is not ideal and relies on trying to match the metric instance iname to a cgroup id. Since iname format varies from one container solution to another, Vector has to try a few known formats and hopefully get a match. If its not a known format (your case), there will be no matches, and no data for the widgets.

Ideally we should have a way of resolving the cgroup id from the metric iname, or have PCP return the cgroup id as the iname for all cgroup.* metric instances.

If you're still facing the same issue, please open another bug and paste the output of the following command:

pminfo --fetch cgroup.cpuacct.usage